class: center, middle, inverse, title-slide # APSTA-GE 2003: Intermediate Quantitative Methods ## Lab Section 003, Week 13 ### New York University ### 12/01/2020 --- ## Reminders - Assignment 7 ([group project](https://drive.google.com/drive/folders/1S0XC8s6-N4OM4X158DmAWjpxtCl_sVRg?usp=sharing)) - [Group assignment](https://docs.google.com/spreadsheets/d/1uZ_ao7dts_wiipddpM9wz_JSYWfBFvcvZ5r6wvqkjTs/edit?usp=sharing) - Due: 12/08/2020 (Friday) 5:00 PM - Sample solutions will be available after due - Office hours - Monday 9:00 - 10:00am (EST) - Wednesday 12:30 - 1:30pm (EST) - Office hour Zoom link [HERE](https://nyu.zoom.us/j/9985119253) --- ## Today's Topics - Bias-Variance Tradeoff - Regularization - Lasso - Ridge - Elastic Net --- class: inverse, center, middle # Bias-Variance Tradeoff --- ## Recall on Regression Models One of the purposes is to predict (estimate) dependent variables using one or many independent variables (predictors), assuming a linear relationship. We fit a regression model to sample data in order to predict population parameters. **Predict population parameter using ordinary least squares approach:** `$$Y = \beta_0 + \beta_1 \cdot X_i + \varepsilon$$` `$$\hat{\beta_1} = ||(y - \beta_0) - X\hat{\beta_1}||^2 = \mathcal{argmin}\sum_{i=1}^n \left( (y_i - \beta_0) - x'_i\hat{\beta_1} \right)^2$$` We minimize the above loss function to esimate the population parameter `\(\beta_1\)`. (Ordinary Least Squares) *Demo*: [Linear Regression Simulator](https://a3sr-2003-iqm.shinyapps.io/2003iqm/) --- ## Loss Function Loss function: assuming the intercept is zero, `$$\mathcal{L}_\text{OLS}(\hat{\beta}) = \sum_{i=1}^n \left(y_i - x'_i\hat{\beta} \right)^2$$` <div id="htmlwidget-516ea3a984ccc53aec07" style="width:720px;height:432px;" class="plotly html-widget"></div> <script type="application/json" data-for="htmlwidget-516ea3a984ccc53aec07">{"x":{"data":[{"x":[9.40896889741563,10.0265943690162,8.48344690291814,8.63734665070419,11.1784891560316,9.06584868032663,11.323605646212,10.6249177899447,9.9542770442939,8.99587942432772,9.17156676333604,9.65164827482702,8.46170660291295,9.74443475453505,8.85005496729245,10.0123269676795,9.77703045925238,10.8877716479113,9.40784472012325,9.34428188141911,9.31748237778498,9.98414180727173,9.55739521464496,10.3525574994488,10.0731705822914,10.0071588003806,9.81239988939775,9.23429935447461,9.77894317925504,9.0164114125298,8.89571595784903,9.06184978582164,10.6786242441826,8.42250213470517,9.13006154177436,10.4846770453571,9.81394730135561,11.5455547001837,9.3886199298718,9.65224351266365,8.36348369446058,10.0203814401,10.8917426801678,9.12725031674775,10.8900508298395,9.65612565387065,7.813218631672,10.880058181492,10.7238565561464,10.2198526802605],"y":[19.607808368832,19.8231948379548,16.1483913232085,17.774427458694,22.5161706601493,18.6743237888309,22.4905662382413,21.6886289005321,21.3964246838174,18.0519239464509,17.4941206587398,21.6429896068613,16.8022102536185,17.5386621380125,18.2388214248934,21.718168752187,18.7630926980025,20.7002826957349,18.2078143178746,19.4429654234271,19.0884408180852,19.8448499340972,18.3516936070062,20.9333850869522,21.2658030810532,20.170890784389,18.9360276447131,18.9215483130989,18.4903397084601,18.4343879738041,17.7266582247708,18.4391958617438,20.7515669337006,15.9374195778284,20.5217320593308,20.3661273825149,18.3300325071039,23.5975606031312,17.9239055973549,17.7984552359788,17.9292962883137,19.0128974944754,22.7217553181207,17.7113459750819,22.2931967847394,18.959660425222,16.9529705499491,20.6199645156089,22.8608230502522,19.8375265766657],"text":["x: 9.408969<br />y: 19.60781","x: 10.026594<br />y: 19.82319","x: 8.483447<br />y: 16.14839","x: 8.637347<br />y: 17.77443","x: 11.178489<br />y: 22.51617","x: 9.065849<br />y: 18.67432","x: 11.323606<br />y: 22.49057","x: 10.624918<br />y: 21.68863","x: 9.954277<br />y: 21.39642","x: 8.995879<br />y: 18.05192","x: 9.171567<br />y: 17.49412","x: 9.651648<br />y: 21.64299","x: 8.461707<br />y: 16.80221","x: 9.744435<br />y: 17.53866","x: 8.850055<br />y: 18.23882","x: 10.012327<br />y: 21.71817","x: 9.777030<br />y: 18.76309","x: 10.887772<br />y: 20.70028","x: 9.407845<br />y: 18.20781","x: 9.344282<br />y: 19.44297","x: 9.317482<br />y: 19.08844","x: 9.984142<br />y: 19.84485","x: 9.557395<br />y: 18.35169","x: 10.352557<br />y: 20.93339","x: 10.073171<br />y: 21.26580","x: 10.007159<br />y: 20.17089","x: 9.812400<br />y: 18.93603","x: 9.234299<br />y: 18.92155","x: 9.778943<br />y: 18.49034","x: 9.016411<br />y: 18.43439","x: 8.895716<br />y: 17.72666","x: 9.061850<br />y: 18.43920","x: 10.678624<br />y: 20.75157","x: 8.422502<br />y: 15.93742","x: 9.130062<br />y: 20.52173","x: 10.484677<br />y: 20.36613","x: 9.813947<br />y: 18.33003","x: 11.545555<br />y: 23.59756","x: 9.388620<br />y: 17.92391","x: 9.652244<br />y: 17.79846","x: 8.363484<br />y: 17.92930","x: 10.020381<br />y: 19.01290","x: 10.891743<br />y: 22.72176","x: 9.127250<br />y: 17.71135","x: 10.890051<br />y: 22.29320","x: 9.656126<br />y: 18.95966","x: 7.813219<br />y: 16.95297","x: 10.880058<br />y: 20.61996","x: 10.723857<br />y: 22.86082","x: 10.219853<br />y: 19.83753"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(0,0,0,1)","opacity":1,"size":5.66929133858268,"symbol":"circle","line":{"width":1.88976377952756,"color":"rgba(0,0,0,1)"}},"hoveron":"points","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[7.813218631672,7.8604633920329,7.90770815239381,7.95495291275472,8.00219767311563,8.04944243347653,8.09668719383744,8.14393195419835,8.19117671455925,8.23842147492016,8.28566623528107,8.33291099564198,8.38015575600288,8.42740051636379,8.4746452767247,8.5218900370856,8.56913479744651,8.61637955780742,8.66362431816833,8.71086907852923,8.75811383889014,8.80535859925105,8.85260335961195,8.89984811997286,8.94709288033377,8.99433764069468,9.04158240105558,9.08882716141649,9.1360719217774,9.1833166821383,9.23056144249921,9.27780620286012,9.32505096322103,9.37229572358193,9.41954048394284,9.46678524430375,9.51403000466465,9.56127476502556,9.60851952538647,9.65576428574738,9.70300904610828,9.75025380646919,9.7974985668301,9.844743327191,9.89198808755191,9.93923284791282,9.98647760827373,10.0337223686346,10.0809671289955,10.1282118893564,10.1754566497174,10.2227014100783,10.2699461704392,10.3171909308001,10.364435691161,10.4116804515219,10.4589252118828,10.5061699722437,10.5534147326046,10.6006594929655,10.6479042533264,10.6951490136873,10.7423937740482,10.7896385344091,10.8368832947701,10.884128055131,10.9313728154919,10.9786175758528,11.0258623362137,11.0731070965746,11.1203518569355,11.1675966172964,11.2148413776573,11.2620861380182,11.3093308983791,11.35657565874,11.4038204191009,11.4510651794618,11.4983099398228,11.5455547001837],"y":[15.8495297661481,15.9393972410142,16.0292647158804,16.1191321907466,16.2089996656128,16.298867140479,16.3887346153452,16.4786020902114,16.5684695650776,16.6583370399438,16.74820451481,16.8380719896762,16.9279394645424,17.0178069394085,17.1076744142747,17.1975418891409,17.2874093640071,17.3772768388733,17.4671443137395,17.5570117886057,17.6468792634719,17.7367467383381,17.8266142132043,17.9164816880705,18.0063491629367,18.0962166378028,18.186084112669,18.2759515875352,18.3658190624014,18.4556865372676,18.5455540121338,18.635421487,18.7252889618662,18.8151564367324,18.9050239115986,18.9948913864648,19.084758861331,19.1746263361971,19.2644938110633,19.3543612859295,19.4442287607957,19.5340962356619,19.6239637105281,19.7138311853943,19.8036986602605,19.8935661351267,19.9834336099929,20.0733010848591,20.1631685597252,20.2530360345914,20.3429035094576,20.4327709843238,20.52263845919,20.6125059340562,20.7023734089224,20.7922408837886,20.8821083586548,20.971975833521,21.0618433083872,21.1517107832534,21.2415782581195,21.3314457329857,21.4213132078519,21.5111806827181,21.6010481575843,21.6909156324505,21.7807831073167,21.8706505821829,21.9605180570491,22.0503855319153,22.1402530067815,22.2301204816477,22.3199879565138,22.40985543138,22.4997229062462,22.5895903811124,22.6794578559786,22.7693253308448,22.859192805711,22.9490602805772],"text":["x: 7.813219<br />y: 15.84953","x: 7.860463<br />y: 15.93940","x: 7.907708<br />y: 16.02926","x: 7.954953<br />y: 16.11913","x: 8.002198<br />y: 16.20900","x: 8.049442<br />y: 16.29887","x: 8.096687<br />y: 16.38873","x: 8.143932<br />y: 16.47860","x: 8.191177<br />y: 16.56847","x: 8.238421<br />y: 16.65834","x: 8.285666<br />y: 16.74820","x: 8.332911<br />y: 16.83807","x: 8.380156<br />y: 16.92794","x: 8.427401<br />y: 17.01781","x: 8.474645<br />y: 17.10767","x: 8.521890<br />y: 17.19754","x: 8.569135<br />y: 17.28741","x: 8.616380<br />y: 17.37728","x: 8.663624<br />y: 17.46714","x: 8.710869<br />y: 17.55701","x: 8.758114<br />y: 17.64688","x: 8.805359<br />y: 17.73675","x: 8.852603<br />y: 17.82661","x: 8.899848<br />y: 17.91648","x: 8.947093<br />y: 18.00635","x: 8.994338<br />y: 18.09622","x: 9.041582<br />y: 18.18608","x: 9.088827<br />y: 18.27595","x: 9.136072<br />y: 18.36582","x: 9.183317<br />y: 18.45569","x: 9.230561<br />y: 18.54555","x: 9.277806<br />y: 18.63542","x: 9.325051<br />y: 18.72529","x: 9.372296<br />y: 18.81516","x: 9.419540<br />y: 18.90502","x: 9.466785<br />y: 18.99489","x: 9.514030<br />y: 19.08476","x: 9.561275<br />y: 19.17463","x: 9.608520<br />y: 19.26449","x: 9.655764<br />y: 19.35436","x: 9.703009<br />y: 19.44423","x: 9.750254<br />y: 19.53410","x: 9.797499<br />y: 19.62396","x: 9.844743<br />y: 19.71383","x: 9.891988<br />y: 19.80370","x: 9.939233<br />y: 19.89357","x: 9.986478<br />y: 19.98343","x: 10.033722<br />y: 20.07330","x: 10.080967<br />y: 20.16317","x: 10.128212<br />y: 20.25304","x: 10.175457<br />y: 20.34290","x: 10.222701<br />y: 20.43277","x: 10.269946<br />y: 20.52264","x: 10.317191<br />y: 20.61251","x: 10.364436<br />y: 20.70237","x: 10.411680<br />y: 20.79224","x: 10.458925<br />y: 20.88211","x: 10.506170<br />y: 20.97198","x: 10.553415<br />y: 21.06184","x: 10.600659<br />y: 21.15171","x: 10.647904<br />y: 21.24158","x: 10.695149<br />y: 21.33145","x: 10.742394<br />y: 21.42131","x: 10.789639<br />y: 21.51118","x: 10.836883<br />y: 21.60105","x: 10.884128<br />y: 21.69092","x: 10.931373<br />y: 21.78078","x: 10.978618<br />y: 21.87065","x: 11.025862<br />y: 21.96052","x: 11.073107<br />y: 22.05039","x: 11.120352<br />y: 22.14025","x: 11.167597<br />y: 22.23012","x: 11.214841<br />y: 22.31999","x: 11.262086<br />y: 22.40986","x: 11.309331<br />y: 22.49972","x: 11.356576<br />y: 22.58959","x: 11.403820<br />y: 22.67946","x: 11.451065<br />y: 22.76933","x: 11.498310<br />y: 22.85919","x: 11.545555<br />y: 22.94906"],"type":"scatter","mode":"lines","name":"fitted values","line":{"width":3.77952755905512,"color":"rgba(51,102,255,1)","dash":"solid"},"hoveron":"points","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[7.62660182824641,11.7321715036092],"y":[19.4690089217922,19.4690089217922],"text":"yintercept: 19.46901","type":"scatter","mode":"lines","line":{"width":1.88976377952756,"color":"rgba(255,0,0,1)","dash":"dash"},"hoveron":"points","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null}],"layout":{"margin":{"t":28.1765601217656,"r":7.30593607305936,"b":42.130898021309,"l":37.2602739726027},"plot_bgcolor":"rgba(235,235,235,1)","paper_bgcolor":"rgba(255,255,255,1)","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"xaxis":{"domain":[0,1],"automargin":true,"type":"linear","autorange":false,"range":[7.62660182824641,11.7321715036092],"tickmode":"array","ticktext":["8","9","10","11"],"tickvals":[8,9,10,11],"categoryorder":"array","categoryarray":["8","9","10","11"],"nticks":null,"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":-0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(255,255,255,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"y","title":{"text":"x","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187}},"hoverformat":".2f"},"yaxis":{"domain":[0,1],"automargin":true,"type":"linear","autorange":false,"range":[15.4621282242989,23.9849621449803],"tickmode":"array","ticktext":["16","18","20","22"],"tickvals":[16,18,20,22],"categoryorder":"array","categoryarray":["16","18","20","22"],"nticks":null,"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":-0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(255,255,255,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"x","title":{"text":"y","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187}},"hoverformat":".2f"},"shapes":[{"type":"rect","fillcolor":null,"line":{"color":null,"width":0,"linetype":[]},"yref":"paper","xref":"paper","x0":0,"x1":1,"y0":0,"y1":1}],"showlegend":false,"legend":{"bgcolor":"rgba(255,255,255,1)","bordercolor":"transparent","borderwidth":1.88976377952756,"font":{"color":"rgba(0,0,0,1)","family":"","size":11.689497716895}},"hovermode":"closest","barmode":"relative"},"config":{"doubleClick":"reset","showSendToCloud":false},"source":"A","attrs":{"15ce325b84f39":{"x":{},"y":{},"type":"scatter"},"15ce367593a87":{"x":{},"y":{}},"15ce31311aa13":{"yintercept":{}}},"cur_data":"15ce325b84f39","visdat":{"15ce325b84f39":["function (y) ","x"],"15ce367593a87":["function (y) ","x"],"15ce31311aa13":["function (y) ","x"]},"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.2,"selected":{"opacity":1},"debounce":0},"shinyEvents":["plotly_hover","plotly_click","plotly_selected","plotly_relayout","plotly_brushed","plotly_brushing","plotly_clickannotation","plotly_doubleclick","plotly_deselect","plotly_afterplot","plotly_sunburstclick"],"base_url":"https://plot.ly"},"evals":[],"jsHooks":[]}</script> --- ## Bias and Variance **Bias**: the difference between the population parameter ($\beta$) and the estimated parameter ($\hat{\beta}$): `$$\textbf{Bias}(\hat{\beta}_\text{OLS}) = \hat{\beta}_\text{OLS} - \beta$$` **Variance**: the spread, or uncertainty, of parameter estimates. `$$\mathcal{X}\hat{\beta} = \mathcal{Y} \\ \hat{\beta} = \frac{\mathcal{Y}}{\mathcal{X}} \\ \textbf{Var}(\hat{\beta}) = \frac{\sigma_{\text{error}}^2}{\mathcal{X}} = \frac{\frac{\varepsilon'\varepsilon}{n - k}}{\mathcal{X}}$$` --- ## Bias and Variance Combinations  --- ## Demo <div id="htmlwidget-8fc40e07354fdc6395f7" style="width:720px;height:432px;" class="plotly html-widget"></div> <script type="application/json" data-for="htmlwidget-8fc40e07354fdc6395f7">{"x":{"data":[{"x":[9.40896889741563,10.0265943690162,8.48344690291814,8.63734665070419,11.1784891560316,9.06584868032663,11.323605646212,10.6249177899447,9.9542770442939,8.99587942432772,9.17156676333604,9.65164827482702,8.46170660291295,9.74443475453505,8.85005496729245,10.0123269676795,9.77703045925238,10.8877716479113,9.40784472012325,9.34428188141911,9.31748237778498,9.98414180727173,9.55739521464496,10.3525574994488,10.0731705822914,10.0071588003806,9.81239988939775,9.23429935447461,9.77894317925504,9.0164114125298,8.89571595784903,9.06184978582164,10.6786242441826,8.42250213470517,9.13006154177436,10.4846770453571,9.81394730135561,11.5455547001837,9.3886199298718,9.65224351266365,8.36348369446058,10.0203814401,10.8917426801678,9.12725031674775,10.8900508298395,9.65612565387065,7.813218631672,10.880058181492,10.7238565561464,10.2198526802605],"y":[19.607808368832,19.8231948379548,16.1483913232085,17.774427458694,22.5161706601493,18.6743237888309,22.4905662382413,21.6886289005321,21.3964246838174,18.0519239464509,17.4941206587398,21.6429896068613,16.8022102536185,17.5386621380125,18.2388214248934,21.718168752187,18.7630926980025,20.7002826957349,18.2078143178746,19.4429654234271,19.0884408180852,19.8448499340972,18.3516936070062,20.9333850869522,21.2658030810532,20.170890784389,18.9360276447131,18.9215483130989,18.4903397084601,18.4343879738041,17.7266582247708,18.4391958617438,20.7515669337006,15.9374195778284,20.5217320593308,20.3661273825149,18.3300325071039,23.5975606031312,17.9239055973549,17.7984552359788,17.9292962883137,19.0128974944754,22.7217553181207,17.7113459750819,22.2931967847394,18.959660425222,16.9529705499491,20.6199645156089,22.8608230502522,19.8375265766657],"text":["x: 9.408969<br />y: 19.60781","x: 10.026594<br />y: 19.82319","x: 8.483447<br />y: 16.14839","x: 8.637347<br />y: 17.77443","x: 11.178489<br />y: 22.51617","x: 9.065849<br />y: 18.67432","x: 11.323606<br />y: 22.49057","x: 10.624918<br />y: 21.68863","x: 9.954277<br />y: 21.39642","x: 8.995879<br />y: 18.05192","x: 9.171567<br />y: 17.49412","x: 9.651648<br />y: 21.64299","x: 8.461707<br />y: 16.80221","x: 9.744435<br />y: 17.53866","x: 8.850055<br />y: 18.23882","x: 10.012327<br />y: 21.71817","x: 9.777030<br />y: 18.76309","x: 10.887772<br />y: 20.70028","x: 9.407845<br />y: 18.20781","x: 9.344282<br />y: 19.44297","x: 9.317482<br />y: 19.08844","x: 9.984142<br />y: 19.84485","x: 9.557395<br />y: 18.35169","x: 10.352557<br />y: 20.93339","x: 10.073171<br />y: 21.26580","x: 10.007159<br />y: 20.17089","x: 9.812400<br />y: 18.93603","x: 9.234299<br />y: 18.92155","x: 9.778943<br />y: 18.49034","x: 9.016411<br />y: 18.43439","x: 8.895716<br />y: 17.72666","x: 9.061850<br />y: 18.43920","x: 10.678624<br />y: 20.75157","x: 8.422502<br />y: 15.93742","x: 9.130062<br />y: 20.52173","x: 10.484677<br />y: 20.36613","x: 9.813947<br />y: 18.33003","x: 11.545555<br />y: 23.59756","x: 9.388620<br />y: 17.92391","x: 9.652244<br />y: 17.79846","x: 8.363484<br />y: 17.92930","x: 10.020381<br />y: 19.01290","x: 10.891743<br />y: 22.72176","x: 9.127250<br />y: 17.71135","x: 10.890051<br />y: 22.29320","x: 9.656126<br />y: 18.95966","x: 7.813219<br />y: 16.95297","x: 10.880058<br />y: 20.61996","x: 10.723857<br />y: 22.86082","x: 10.219853<br />y: 19.83753"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(0,0,0,1)","opacity":1,"size":5.66929133858268,"symbol":"circle","line":{"width":1.88976377952756,"color":"rgba(0,0,0,1)"}},"hoveron":"points","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[7.813218631672,7.8604633920329,7.90770815239381,7.95495291275472,8.00219767311563,8.04944243347653,8.09668719383744,8.14393195419835,8.19117671455925,8.23842147492016,8.28566623528107,8.33291099564198,8.38015575600288,8.42740051636379,8.4746452767247,8.5218900370856,8.56913479744651,8.61637955780742,8.66362431816833,8.71086907852923,8.75811383889014,8.80535859925105,8.85260335961195,8.89984811997286,8.94709288033377,8.99433764069468,9.04158240105558,9.08882716141649,9.1360719217774,9.1833166821383,9.23056144249921,9.27780620286012,9.32505096322103,9.37229572358193,9.41954048394284,9.46678524430375,9.51403000466465,9.56127476502556,9.60851952538647,9.65576428574738,9.70300904610828,9.75025380646919,9.7974985668301,9.844743327191,9.89198808755191,9.93923284791282,9.98647760827373,10.0337223686346,10.0809671289955,10.1282118893564,10.1754566497174,10.2227014100783,10.2699461704392,10.3171909308001,10.364435691161,10.4116804515219,10.4589252118828,10.5061699722437,10.5534147326046,10.6006594929655,10.6479042533264,10.6951490136873,10.7423937740482,10.7896385344091,10.8368832947701,10.884128055131,10.9313728154919,10.9786175758528,11.0258623362137,11.0731070965746,11.1203518569355,11.1675966172964,11.2148413776573,11.2620861380182,11.3093308983791,11.35657565874,11.4038204191009,11.4510651794618,11.4983099398228,11.5455547001837],"y":[15.8495297661481,15.9393972410142,16.0292647158804,16.1191321907466,16.2089996656128,16.298867140479,16.3887346153452,16.4786020902114,16.5684695650776,16.6583370399438,16.74820451481,16.8380719896762,16.9279394645424,17.0178069394085,17.1076744142747,17.1975418891409,17.2874093640071,17.3772768388733,17.4671443137395,17.5570117886057,17.6468792634719,17.7367467383381,17.8266142132043,17.9164816880705,18.0063491629367,18.0962166378028,18.186084112669,18.2759515875352,18.3658190624014,18.4556865372676,18.5455540121338,18.635421487,18.7252889618662,18.8151564367324,18.9050239115986,18.9948913864648,19.084758861331,19.1746263361971,19.2644938110633,19.3543612859295,19.4442287607957,19.5340962356619,19.6239637105281,19.7138311853943,19.8036986602605,19.8935661351267,19.9834336099929,20.0733010848591,20.1631685597252,20.2530360345914,20.3429035094576,20.4327709843238,20.52263845919,20.6125059340562,20.7023734089224,20.7922408837886,20.8821083586548,20.971975833521,21.0618433083872,21.1517107832534,21.2415782581195,21.3314457329857,21.4213132078519,21.5111806827181,21.6010481575843,21.6909156324505,21.7807831073167,21.8706505821829,21.9605180570491,22.0503855319153,22.1402530067815,22.2301204816477,22.3199879565138,22.40985543138,22.4997229062462,22.5895903811124,22.6794578559786,22.7693253308448,22.859192805711,22.9490602805772],"text":["x: 7.813219<br />y: 15.84953","x: 7.860463<br />y: 15.93940","x: 7.907708<br />y: 16.02926","x: 7.954953<br />y: 16.11913","x: 8.002198<br />y: 16.20900","x: 8.049442<br />y: 16.29887","x: 8.096687<br />y: 16.38873","x: 8.143932<br />y: 16.47860","x: 8.191177<br />y: 16.56847","x: 8.238421<br />y: 16.65834","x: 8.285666<br />y: 16.74820","x: 8.332911<br />y: 16.83807","x: 8.380156<br />y: 16.92794","x: 8.427401<br />y: 17.01781","x: 8.474645<br />y: 17.10767","x: 8.521890<br />y: 17.19754","x: 8.569135<br />y: 17.28741","x: 8.616380<br />y: 17.37728","x: 8.663624<br />y: 17.46714","x: 8.710869<br />y: 17.55701","x: 8.758114<br />y: 17.64688","x: 8.805359<br />y: 17.73675","x: 8.852603<br />y: 17.82661","x: 8.899848<br />y: 17.91648","x: 8.947093<br />y: 18.00635","x: 8.994338<br />y: 18.09622","x: 9.041582<br />y: 18.18608","x: 9.088827<br />y: 18.27595","x: 9.136072<br />y: 18.36582","x: 9.183317<br />y: 18.45569","x: 9.230561<br />y: 18.54555","x: 9.277806<br />y: 18.63542","x: 9.325051<br />y: 18.72529","x: 9.372296<br />y: 18.81516","x: 9.419540<br />y: 18.90502","x: 9.466785<br />y: 18.99489","x: 9.514030<br />y: 19.08476","x: 9.561275<br />y: 19.17463","x: 9.608520<br />y: 19.26449","x: 9.655764<br />y: 19.35436","x: 9.703009<br />y: 19.44423","x: 9.750254<br />y: 19.53410","x: 9.797499<br />y: 19.62396","x: 9.844743<br />y: 19.71383","x: 9.891988<br />y: 19.80370","x: 9.939233<br />y: 19.89357","x: 9.986478<br />y: 19.98343","x: 10.033722<br />y: 20.07330","x: 10.080967<br />y: 20.16317","x: 10.128212<br />y: 20.25304","x: 10.175457<br />y: 20.34290","x: 10.222701<br />y: 20.43277","x: 10.269946<br />y: 20.52264","x: 10.317191<br />y: 20.61251","x: 10.364436<br />y: 20.70237","x: 10.411680<br />y: 20.79224","x: 10.458925<br />y: 20.88211","x: 10.506170<br />y: 20.97198","x: 10.553415<br />y: 21.06184","x: 10.600659<br />y: 21.15171","x: 10.647904<br />y: 21.24158","x: 10.695149<br />y: 21.33145","x: 10.742394<br />y: 21.42131","x: 10.789639<br />y: 21.51118","x: 10.836883<br />y: 21.60105","x: 10.884128<br />y: 21.69092","x: 10.931373<br />y: 21.78078","x: 10.978618<br />y: 21.87065","x: 11.025862<br />y: 21.96052","x: 11.073107<br />y: 22.05039","x: 11.120352<br />y: 22.14025","x: 11.167597<br />y: 22.23012","x: 11.214841<br />y: 22.31999","x: 11.262086<br />y: 22.40986","x: 11.309331<br />y: 22.49972","x: 11.356576<br />y: 22.58959","x: 11.403820<br />y: 22.67946","x: 11.451065<br />y: 22.76933","x: 11.498310<br />y: 22.85919","x: 11.545555<br />y: 22.94906"],"type":"scatter","mode":"lines","name":"fitted values","line":{"width":3.77952755905512,"color":"rgba(51,102,255,1)","dash":"solid"},"hoveron":"points","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[7.62660182824641,11.7321715036092],"y":[19.4690089217922,19.4690089217922],"text":"yintercept: 19.46901","type":"scatter","mode":"lines","line":{"width":1.88976377952756,"color":"rgba(255,0,0,1)","dash":"dash"},"hoveron":"points","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[7.62660182824641,11.7321715036092],"y":[19.6266018282464,23.7321715036092],"text":"intercept: 12<br />slope: 1","type":"scatter","mode":"lines","line":{"width":1.88976377952756,"color":"rgba(255,255,0,1)","dash":"solid"},"hoveron":"points","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[7.813218631672,8.36348369446058,8.42250213470517,8.46170660291295,8.48344690291814,8.63734665070419,8.85005496729245,8.89571595784903,8.99587942432772,9.0164114125298,9.06184978582164,9.06584868032663,9.12725031674775,9.13006154177436,9.17156676333604,9.23429935447461,9.31748237778498,9.34428188141911,9.3886199298718,9.40784472012325,9.40896889741563,9.55739521464496,9.65164827482702,9.65224351266365,9.65612565387065,9.74443475453505,9.77703045925238,9.77894317925504,9.81239988939775,9.81394730135561,9.9542770442939,9.98414180727173,10.0071588003806,10.0123269676795,10.0203814401,10.0265943690162,10.0731705822914,10.2198526802605,10.3525574994488,10.4846770453571,10.6249177899447,10.6786242441826,10.7238565561464,10.880058181492,10.8877716479113,10.8900508298395,10.8917426801678,11.1784891560316,11.323605646212,11.5455547001837],"y":[16.9529705499491,17.9292962883137,15.9374195778284,16.8022102536185,16.1483913232085,17.774427458694,18.2388214248934,17.7266582247708,18.0519239464509,18.4343879738041,18.4391958617438,18.6743237888309,17.7113459750819,20.5217320593308,17.4941206587398,18.9215483130989,19.0884408180852,19.4429654234271,17.9239055973549,18.2078143178746,19.607808368832,18.3516936070062,21.6429896068613,17.7984552359788,18.959660425222,17.5386621380125,18.7630926980025,18.4903397084601,18.9360276447131,18.3300325071039,21.3964246838174,19.8448499340972,20.170890784389,21.718168752187,19.0128974944754,19.8231948379548,21.2658030810532,19.8375265766657,20.9333850869522,20.3661273825149,21.6886289005321,20.7515669337006,22.8608230502522,20.6199645156089,20.7002826957349,22.2931967847394,22.7217553181207,22.5161706601493,22.4905662382413,23.5975606031312],"text":["x: 7.813219<br />y: 16.95297","x: 8.363484<br />y: 17.92930","x: 8.422502<br />y: 15.93742","x: 8.461707<br />y: 16.80221","x: 8.483447<br />y: 16.14839","x: 8.637347<br />y: 17.77443","x: 8.850055<br />y: 18.23882","x: 8.895716<br />y: 17.72666","x: 8.995879<br />y: 18.05192","x: 9.016411<br />y: 18.43439","x: 9.061850<br />y: 18.43920","x: 9.065849<br />y: 18.67432","x: 9.127250<br />y: 17.71135","x: 9.130062<br />y: 20.52173","x: 9.171567<br />y: 17.49412","x: 9.234299<br />y: 18.92155","x: 9.317482<br />y: 19.08844","x: 9.344282<br />y: 19.44297","x: 9.388620<br />y: 17.92391","x: 9.407845<br />y: 18.20781","x: 9.408969<br />y: 19.60781","x: 9.557395<br />y: 18.35169","x: 9.651648<br />y: 21.64299","x: 9.652244<br />y: 17.79846","x: 9.656126<br />y: 18.95966","x: 9.744435<br />y: 17.53866","x: 9.777030<br />y: 18.76309","x: 9.778943<br />y: 18.49034","x: 9.812400<br />y: 18.93603","x: 9.813947<br />y: 18.33003","x: 9.954277<br />y: 21.39642","x: 9.984142<br />y: 19.84485","x: 10.007159<br />y: 20.17089","x: 10.012327<br />y: 21.71817","x: 10.020381<br />y: 19.01290","x: 10.026594<br />y: 19.82319","x: 10.073171<br />y: 21.26580","x: 10.219853<br />y: 19.83753","x: 10.352557<br />y: 20.93339","x: 10.484677<br />y: 20.36613","x: 10.624918<br />y: 21.68863","x: 10.678624<br />y: 20.75157","x: 10.723857<br />y: 22.86082","x: 10.880058<br />y: 20.61996","x: 10.887772<br />y: 20.70028","x: 10.890051<br />y: 22.29320","x: 10.891743<br />y: 22.72176","x: 11.178489<br />y: 22.51617","x: 11.323606<br />y: 22.49057","x: 11.545555<br />y: 23.59756"],"type":"scatter","mode":"lines","line":{"width":1.88976377952756,"color":"rgba(169,169,169,1)","dash":"solid"},"hoveron":"points","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null}],"layout":{"margin":{"t":28.1765601217656,"r":7.30593607305936,"b":42.130898021309,"l":37.2602739726027},"plot_bgcolor":"rgba(235,235,235,1)","paper_bgcolor":"rgba(255,255,255,1)","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"xaxis":{"domain":[0,1],"automargin":true,"type":"linear","autorange":false,"range":[7.62660182824641,11.7321715036092],"tickmode":"array","ticktext":["8","9","10","11"],"tickvals":[8,9,10,11],"categoryorder":"array","categoryarray":["8","9","10","11"],"nticks":null,"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":-0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(255,255,255,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"y","title":{"text":"x","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187}},"hoverformat":".2f"},"yaxis":{"domain":[0,1],"automargin":true,"type":"linear","autorange":false,"range":[15.4621282242989,23.9849621449803],"tickmode":"array","ticktext":["16","18","20","22"],"tickvals":[16,18,20,22],"categoryorder":"array","categoryarray":["16","18","20","22"],"nticks":null,"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":-0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(255,255,255,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"x","title":{"text":"y","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187}},"hoverformat":".2f"},"shapes":[{"type":"rect","fillcolor":null,"line":{"color":null,"width":0,"linetype":[]},"yref":"paper","xref":"paper","x0":0,"x1":1,"y0":0,"y1":1}],"showlegend":false,"legend":{"bgcolor":"rgba(255,255,255,1)","bordercolor":"transparent","borderwidth":1.88976377952756,"font":{"color":"rgba(0,0,0,1)","family":"","size":11.689497716895}},"hovermode":"closest","barmode":"relative"},"config":{"doubleClick":"reset","showSendToCloud":false},"source":"A","attrs":{"15ce31c726736":{"x":{},"y":{},"type":"scatter"},"15ce31ad618d1":{"x":{},"y":{}},"15ce35df74f1a":{"yintercept":{}},"15ce3176a6428":{"intercept":{},"slope":{}},"15ce349d58a1a":{"x":{},"y":{}}},"cur_data":"15ce31c726736","visdat":{"15ce31c726736":["function (y) ","x"],"15ce31ad618d1":["function (y) ","x"],"15ce35df74f1a":["function (y) ","x"],"15ce3176a6428":["function (y) ","x"],"15ce349d58a1a":["function (y) ","x"]},"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.2,"selected":{"opacity":1},"debounce":0},"shinyEvents":["plotly_hover","plotly_click","plotly_selected","plotly_relayout","plotly_brushed","plotly_brushing","plotly_clickannotation","plotly_doubleclick","plotly_deselect","plotly_afterplot","plotly_sunburstclick"],"base_url":"https://plot.ly"},"evals":[],"jsHooks":[]}</script> --- ## Bias-Variance Tradeoff **Overfit**: model is too complex and contains too many noises. **Underfit**: model is too simple and does not reflect features.  --- ## Demo <div id="htmlwidget-dc498c8b3c8ac0cbec08" style="width:720px;height:432px;" class="plotly html-widget"></div> <script type="application/json" data-for="htmlwidget-dc498c8b3c8ac0cbec08">{"x":{"data":[{"x":[9.40896889741563,10.0265943690162,8.48344690291814,8.63734665070419,11.1784891560316,9.06584868032663,11.323605646212,10.6249177899447,9.9542770442939,8.99587942432772,9.17156676333604,9.65164827482702,8.46170660291295,9.74443475453505,8.85005496729245,10.0123269676795,9.77703045925238,10.8877716479113,9.40784472012325,9.34428188141911,9.31748237778498,9.98414180727173,9.55739521464496,10.3525574994488,10.0731705822914,10.0071588003806,9.81239988939775,9.23429935447461,9.77894317925504,9.0164114125298,8.89571595784903,9.06184978582164,10.6786242441826,8.42250213470517,9.13006154177436,10.4846770453571,9.81394730135561,11.5455547001837,9.3886199298718,9.65224351266365,8.36348369446058,10.0203814401,10.8917426801678,9.12725031674775,10.8900508298395,9.65612565387065,7.813218631672,10.880058181492,10.7238565561464,10.2198526802605],"y":[19.607808368832,19.8231948379548,16.1483913232085,17.774427458694,22.5161706601493,18.6743237888309,22.4905662382413,21.6886289005321,21.3964246838174,18.0519239464509,17.4941206587398,21.6429896068613,16.8022102536185,17.5386621380125,18.2388214248934,21.718168752187,18.7630926980025,20.7002826957349,18.2078143178746,19.4429654234271,19.0884408180852,19.8448499340972,18.3516936070062,20.9333850869522,21.2658030810532,20.170890784389,18.9360276447131,18.9215483130989,18.4903397084601,18.4343879738041,17.7266582247708,18.4391958617438,20.7515669337006,15.9374195778284,20.5217320593308,20.3661273825149,18.3300325071039,23.5975606031312,17.9239055973549,17.7984552359788,17.9292962883137,19.0128974944754,22.7217553181207,17.7113459750819,22.2931967847394,18.959660425222,16.9529705499491,20.6199645156089,22.8608230502522,19.8375265766657],"text":["x: 9.408969<br />y: 19.60781","x: 10.026594<br />y: 19.82319","x: 8.483447<br />y: 16.14839","x: 8.637347<br />y: 17.77443","x: 11.178489<br />y: 22.51617","x: 9.065849<br />y: 18.67432","x: 11.323606<br />y: 22.49057","x: 10.624918<br />y: 21.68863","x: 9.954277<br />y: 21.39642","x: 8.995879<br />y: 18.05192","x: 9.171567<br />y: 17.49412","x: 9.651648<br />y: 21.64299","x: 8.461707<br />y: 16.80221","x: 9.744435<br />y: 17.53866","x: 8.850055<br />y: 18.23882","x: 10.012327<br />y: 21.71817","x: 9.777030<br />y: 18.76309","x: 10.887772<br />y: 20.70028","x: 9.407845<br />y: 18.20781","x: 9.344282<br />y: 19.44297","x: 9.317482<br />y: 19.08844","x: 9.984142<br />y: 19.84485","x: 9.557395<br />y: 18.35169","x: 10.352557<br />y: 20.93339","x: 10.073171<br />y: 21.26580","x: 10.007159<br />y: 20.17089","x: 9.812400<br />y: 18.93603","x: 9.234299<br />y: 18.92155","x: 9.778943<br />y: 18.49034","x: 9.016411<br />y: 18.43439","x: 8.895716<br />y: 17.72666","x: 9.061850<br />y: 18.43920","x: 10.678624<br />y: 20.75157","x: 8.422502<br />y: 15.93742","x: 9.130062<br />y: 20.52173","x: 10.484677<br />y: 20.36613","x: 9.813947<br />y: 18.33003","x: 11.545555<br />y: 23.59756","x: 9.388620<br />y: 17.92391","x: 9.652244<br />y: 17.79846","x: 8.363484<br />y: 17.92930","x: 10.020381<br />y: 19.01290","x: 10.891743<br />y: 22.72176","x: 9.127250<br />y: 17.71135","x: 10.890051<br />y: 22.29320","x: 9.656126<br />y: 18.95966","x: 7.813219<br />y: 16.95297","x: 10.880058<br />y: 20.61996","x: 10.723857<br />y: 22.86082","x: 10.219853<br />y: 19.83753"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(0,0,0,1)","opacity":1,"size":5.66929133858268,"symbol":"circle","line":{"width":1.88976377952756,"color":"rgba(0,0,0,1)"}},"hoveron":"points","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[7.813218631672,8.36348369446058,8.42250213470517,8.46170660291295,8.48344690291814,8.63734665070419,8.85005496729245,8.89571595784903,8.99587942432772,9.0164114125298,9.06184978582164,9.06584868032663,9.12725031674775,9.13006154177436,9.17156676333604,9.23429935447461,9.31748237778498,9.34428188141911,9.3886199298718,9.40784472012325,9.40896889741563,9.55739521464496,9.65164827482702,9.65224351266365,9.65612565387065,9.74443475453505,9.77703045925238,9.77894317925504,9.81239988939775,9.81394730135561,9.9542770442939,9.98414180727173,10.0071588003806,10.0123269676795,10.0203814401,10.0265943690162,10.0731705822914,10.2198526802605,10.3525574994488,10.4846770453571,10.6249177899447,10.6786242441826,10.7238565561464,10.880058181492,10.8877716479113,10.8900508298395,10.8917426801678,11.1784891560316,11.323605646212,11.5455547001837],"y":[16.9529705499491,17.9292962883137,15.9374195778284,16.8022102536185,16.1483913232085,17.774427458694,18.2388214248934,17.7266582247708,18.0519239464509,18.4343879738041,18.4391958617438,18.6743237888309,17.7113459750819,20.5217320593308,17.4941206587398,18.9215483130989,19.0884408180852,19.4429654234271,17.9239055973549,18.2078143178746,19.607808368832,18.3516936070062,21.6429896068613,17.7984552359788,18.959660425222,17.5386621380125,18.7630926980025,18.4903397084601,18.9360276447131,18.3300325071039,21.3964246838174,19.8448499340972,20.170890784389,21.718168752187,19.0128974944754,19.8231948379548,21.2658030810532,19.8375265766657,20.9333850869522,20.3661273825149,21.6886289005321,20.7515669337006,22.8608230502522,20.6199645156089,20.7002826957349,22.2931967847394,22.7217553181207,22.5161706601493,22.4905662382413,23.5975606031312],"text":["x: 7.813219<br />y: 16.95297","x: 8.363484<br />y: 17.92930","x: 8.422502<br />y: 15.93742","x: 8.461707<br />y: 16.80221","x: 8.483447<br />y: 16.14839","x: 8.637347<br />y: 17.77443","x: 8.850055<br />y: 18.23882","x: 8.895716<br />y: 17.72666","x: 8.995879<br />y: 18.05192","x: 9.016411<br />y: 18.43439","x: 9.061850<br />y: 18.43920","x: 9.065849<br />y: 18.67432","x: 9.127250<br />y: 17.71135","x: 9.130062<br />y: 20.52173","x: 9.171567<br />y: 17.49412","x: 9.234299<br />y: 18.92155","x: 9.317482<br />y: 19.08844","x: 9.344282<br />y: 19.44297","x: 9.388620<br />y: 17.92391","x: 9.407845<br />y: 18.20781","x: 9.408969<br />y: 19.60781","x: 9.557395<br />y: 18.35169","x: 9.651648<br />y: 21.64299","x: 9.652244<br />y: 17.79846","x: 9.656126<br />y: 18.95966","x: 9.744435<br />y: 17.53866","x: 9.777030<br />y: 18.76309","x: 9.778943<br />y: 18.49034","x: 9.812400<br />y: 18.93603","x: 9.813947<br />y: 18.33003","x: 9.954277<br />y: 21.39642","x: 9.984142<br />y: 19.84485","x: 10.007159<br />y: 20.17089","x: 10.012327<br />y: 21.71817","x: 10.020381<br />y: 19.01290","x: 10.026594<br />y: 19.82319","x: 10.073171<br />y: 21.26580","x: 10.219853<br />y: 19.83753","x: 10.352557<br />y: 20.93339","x: 10.484677<br />y: 20.36613","x: 10.624918<br />y: 21.68863","x: 10.678624<br />y: 20.75157","x: 10.723857<br />y: 22.86082","x: 10.880058<br />y: 20.61996","x: 10.887772<br />y: 20.70028","x: 10.890051<br />y: 22.29320","x: 10.891743<br />y: 22.72176","x: 11.178489<br />y: 22.51617","x: 11.323606<br />y: 22.49057","x: 11.545555<br />y: 23.59756"],"type":"scatter","mode":"lines","line":{"width":1.88976377952756,"color":"rgba(169,169,169,1)","dash":"solid"},"hoveron":"points","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[7.62660182824641,11.7321715036092],"y":[19.4690089217922,19.4690089217922],"text":"yintercept: 19.46901","type":"scatter","mode":"lines","line":{"width":1.88976377952756,"color":"rgba(255,0,0,1)","dash":"solid"},"hoveron":"points","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null}],"layout":{"margin":{"t":28.1765601217656,"r":7.30593607305936,"b":42.130898021309,"l":37.2602739726027},"plot_bgcolor":"rgba(235,235,235,1)","paper_bgcolor":"rgba(255,255,255,1)","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"xaxis":{"domain":[0,1],"automargin":true,"type":"linear","autorange":false,"range":[7.62660182824641,11.7321715036092],"tickmode":"array","ticktext":["8","9","10","11"],"tickvals":[8,9,10,11],"categoryorder":"array","categoryarray":["8","9","10","11"],"nticks":null,"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":-0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(255,255,255,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"y","title":{"text":"x","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187}},"hoverformat":".2f"},"yaxis":{"domain":[0,1],"automargin":true,"type":"linear","autorange":false,"range":[15.5544125265633,23.9805676543963],"tickmode":"array","ticktext":["16","18","20","22"],"tickvals":[16,18,20,22],"categoryorder":"array","categoryarray":["16","18","20","22"],"nticks":null,"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":-0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(255,255,255,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"x","title":{"text":"y","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187}},"hoverformat":".2f"},"shapes":[{"type":"rect","fillcolor":null,"line":{"color":null,"width":0,"linetype":[]},"yref":"paper","xref":"paper","x0":0,"x1":1,"y0":0,"y1":1}],"showlegend":false,"legend":{"bgcolor":"rgba(255,255,255,1)","bordercolor":"transparent","borderwidth":1.88976377952756,"font":{"color":"rgba(0,0,0,1)","family":"","size":11.689497716895}},"hovermode":"closest","barmode":"relative"},"config":{"doubleClick":"reset","showSendToCloud":false},"source":"A","attrs":{"15ce32b34b2bb":{"x":{},"y":{},"type":"scatter"},"15ce310c22926":{"x":{},"y":{}},"15ce33b1b8662":{"yintercept":{}}},"cur_data":"15ce32b34b2bb","visdat":{"15ce32b34b2bb":["function (y) ","x"],"15ce310c22926":["function (y) ","x"],"15ce33b1b8662":["function (y) ","x"]},"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.2,"selected":{"opacity":1},"debounce":0},"shinyEvents":["plotly_hover","plotly_click","plotly_selected","plotly_relayout","plotly_brushed","plotly_brushing","plotly_clickannotation","plotly_doubleclick","plotly_deselect","plotly_afterplot","plotly_sunburstclick"],"base_url":"https://plot.ly"},"evals":[],"jsHooks":[]}</script> --- class: inverse, center, middle # Regularization --- ## Regularizaton Recall that, in model selection, we started from a simple model called baseline model. For example, we use mean model or simple linear regression: `$$Y_1 = \frac{\sum_{i=1}^n x_i}{n} \\ Y_2 = \beta_0 + \beta_1 X + \varepsilon$$` When we increase model complexity, we increase the risk of overfitting. Therefore, we regularize model variance by introducing a penalty parameter, `\(\lambda\)`, to our loss function: `$$\mathcal{L}(\hat{\beta}) = \sum_{i=1}^n \left(y_i - x'_i\hat{\beta} \right)^2 + \lambda\sum_{j=1}^m\hat{\beta_j}^2$$` --- ## Ridge Regression Solving the function, we get `\(\hat{\beta}\)` as: `$$\hat{\beta} = \frac{\mathcal{Y}}{\mathcal{X} + \lambda} \\ \textbf{Bias}(\hat{\beta}) = \hat{\beta} - \beta \\ \textbf{Var}(\hat{\beta}) = \frac{\sigma_{\text{error}}^2}{\mathcal{X} + \lambda}$$` The `\(\lambda\)` parameter is the regularization penalty: `$$\lambda \to 0, \hat{\beta}_\text{ridge} \to \hat{\beta}_\text{OLS} \\ \lambda \to \infty, \hat{\beta}_\text{ridge} \to 0$$` --- ## Lasso Regression Lasso, or least absolute shrinkage and selection operator, penalizes the sum of absolute coefficients: `$$\mathcal{L}(\hat{\beta}) = \sum_{i=1}^n \left(y_i - x'_i\hat{\beta} \right)^2 + \lambda\sum_{j=1}^m|\hat{\beta_j}| \\ \hat{\beta} = \frac{\mathcal{Y} - \lambda}{\mathcal{X}}$$` --- ## Ridge and Lasso Ridge Regression: (L2 penalty) `$$\mathcal{L}_\text{ridge}(\hat{\beta}) = \sum_{i=1}^n \left(y_i - x'_i\hat{\beta} \right)^2 + \lambda\sum_{j=1}^m\hat{\beta_j}^2$$` Lasso Regression: (L1 penalty) `$$\mathcal{L}_{\text{lasso}}(\hat{\beta}) = \sum_{i=1}^n \left(y_i - x'_i\hat{\beta} \right)^2 + \lambda\sum_{j=1}^m|\hat{\beta_j}|$$` Lasso can set some coefficients to zero. Thus, it can perform variable selection while Ridge cannot. --- ## Elastic Net The combination of L1 (Lasso) and L2 (Ridge) penalties. `$$\mathcal{L}_\text{elastic net} = \frac{\sum_{i=1}^n(y_i - x_i'\hat{\beta})^2}{2n} + \lambda\left(\frac{1-\alpha}{2}\sum_{j=1}^m\hat{\beta_j}^2 + \alpha\sum_{j=1}^m|\hat{\beta_j}|\right),$$` where `\(\alpha\)` is the mixing parameter between ridge ($\alpha = 0$) and lasso ($\alpha = 1$). --- ## Contact Tong Jin - Email: tj1061@nyu.edu - Office Hours - Mondays, 9 - 10am (EST) - Wednesdays, 12:30 - 1:30pm (EST)